However, spurious correlation remains an ongoing challenge, primarily due to the difficulty in precisely detecting these samples.

Thesebiased models suffer from the poor generalization capability when evaluated on unbiased datasets. Existing approaches for debiasing often identify and emphasize those samples withnosuchcorrelation (i.e.,bias-conflicting)without defining the bias type in advance. However, such bias-conflicting samples are significantly scarce in biased datasets, limiting the debiasing capability of these approaches.

Whileprevious work tackles this issue by using explicit labeling on the spuriously correlated attributes or presuming a particular bias type, we instead utilize a cheaper, yet generic form of human knowledge, which can be widely applicable to various types ofbias.

For real-world visual datasets (like facial dataset or ImageNet), the unavailability of strict counterfactual data is a common challenge.

Such error correlations not only affect the model's ability to make robust predictions but also

For this, we utilize an auxiliary contrastive model with the popular heuristic that biased features are learned preferentially during training.